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ABSTRACT 

Several analyses were performed as part of the 
process of validating a recently developed scale to measure the 
attitudes of students toward writing. The Florida Writing Project 
Student Survey is a 25-item measure of writing attitudes designed for 
use with students in grades six through twelve. It was developed to 
assess the attitudes of students who had participated in the Florida 
Writing Project. The evaluation consisted of six analyses for 
reliability, item analysis, correlation between the attitude scale 
and holistic writing scores, and the survey's sensitivity to change. 
Further analyses are recommended as a means of refining the survey 
and providing additional validation information. (DWH) 
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Q VALIDATION OF A WRITING ATTITUDE SCALe 
Introduction 

A critical component of any research and evaluation effort often Involves 
decisions concerning the selection or development of appropriate Instruments 
for measuring variables of Interest to the researcher « In deciding whether a 
measure Is appropriate for a particular purpose » the researcher must examine 
evidence of the instrument's reliability and validity « That is, the 
researcher should want to know if the Instrument is measuring what it xs 
Intended to measure and if it is doing so in a consistent fashion. 

Particularly in the case of newly developed Instruments , part of the 
research process should Include gatheilng and providing reliability and 
validity evidence* Many techniques may be used including both objective 
empirical measures and more subjective means « The purpose of the present 
paj^er is to report the results of several analyses performed as part of the 
process of validating, a recently developed scale to measure the attitudes of 
students toward writing* 

Background 

The Florida Writing Project » a research endeavor conducted at the 
University of Florida , was a program designed to improve the writing skills of 
students at the middle and secondary levels* It was implemented in Januavy of 
1981 at the middle school Xavel and received funding the following year for 
Implementation at the secondary level (McLean, O'Neal, McCurley, Fritchi, 
Giles, & Steele, undated). 

Outing the first year of implementation, the writing attitudes of 
students were measured using 'the Emig-King Writing Attitude Scale, this 
Instfl^nt has been described briefly by McLean et al. (undated) in their 
evalaution report. 



To meet their need for a briefer scale which focused on areas more 
specific to the Writing Project (McLean, 1984), directors Helen Guttinger and 
Chris Morris„ modified significantly an instrument used by the Bay Area Writings- 
Project, titled the Florida Writing Project Student Survey. The survey was . 
designed to be use4 as a measure of writing attitudes. Florida students in 
grades 6 through 12 who participated in the Florida Writing Project made tip 
the three groups in the study. The three groups were based on the training of 
the teachers. They included experimental, inservice, and control groups. In 
addition to a pretest/posttest measure of writing skills using holistic 
scoring, the students were pretested and posttested using tfhe survey. 

Description of the Instrument 
The Florida Writing Project Student Survey contains 25 Likert-type items 
calling for students to respond on a scale from 1 (strongly agree) to 5 
(strongly disagree) to various statements about writing and their own writing 
skills. Among the 25 items, 7 negative statements are loca^^d throughout the 
scale. Scoring of the survey consists of reversing positive statements so 
that "strongly agree" equals a response of 5 and then summing item responses 
to obtain a total score. Thus a high score would indicate a positive attitude 
toward writing and a low score would reflect a negative attitude. A copy of 
the survey is provided as Appendix A. 

Methods 

Evaluation of the scale consisted of six analyses. Two types of 
reliability estimates were obtained. Evidence of internal consistency was 
provided by computing Cronbach's alpha for the pretest and posttest scores 
separately as well as for all scores combined (since research results 
suggested little or ^jo treatment effe on writing skills). Test-retest 
reliability was assessed by obtaining Pearson Product-Moment Correlations 



between pretest and posttest results for control group . students* The 
correlations were obtained for all control group students combined as well as 
for each grade separately* 

Principal components factor analysis was conducted to investigate the 
poasiblllty of underlying subscales. The scree test and the Kaiser criterion 
of eigenvalues greater than 1.0 were used In combination In determining the 
number of factots to extract and rotate. Item analysis was also performed 
which examined the strength of the correlation between each Item and the total 

score. ^' 

A twc^fay analysis of variance repeated measures design was used In p<irt 
to assess the survey's sensitivity to change. Finally, Pearson Product-Moment 
Correlations were computed at each grade between the attitude scale and the 
holistic writing score for pretest, posttest, and gain. 

Results and Discussion 
The internal consistency reliability estimate for the scale based on 
pretest scores was a coefficient alpha of .93 (N - 2154). For posttest 
scores, the coefficient alpha was .84 (N - 1896), and for all scores combined 
it was .83 (N « 4057). Alpha coefficients of this magnitude are indicative of 
a fairly high degree of Internal consistency among Items of the scale. 
Nunnally (1978) has suggested estimates of .80 or above are sufficient for 

basic research purposed. 

Test'retest reliability estimates obtained for the control group at each 
grade separately ranged from .61 at grade 8 to .80 at grade 10. They are 
reported along with the N for each group In Table 1. Correlations were not 
computed for two of the grades because In one case there were no students in 
the control group (grade 6) and In another instance there were too few 
students in the control group (grade 11). While the teat-retest reliability 



for the control group students in all grades combined (N » 371) was .72, the 

ft 

time between testings (almost 9 months) may have been a contributing factor. 
Also, an examination of reliabilities by grades shows somewhat lower 
correlations for the lower grades with correlations reaching .80 for grades 10 
and 12. „ 



Table 1 

Test-Retest Reliability Coefficients for the Cc^ntrol Group by Grade 



Grade 




N 


r 


7 


o 


77 




8 




78 


.61 


9 




45 


.65 


10 




59 


.80 


11 








12 




106 


.79 



Principal components factor analysis using a combination of the scree 
test and Kaiser criterion suggested the possible existence of 4 to 6 factors. 
Varlmax rotation was performed using 4» 5» and 6 factors as Input* Simple 
structure was best approximated using 4 factors and a criterion of .4 
correlation. In this Instance » 20 items loaded on a single factor ^ 2 items 
loaded on two factors » and 3 items did not load on any factor. 

Subjective evaluation of item content within each factor led to tentative 
labels. Items loading on Factor 1 alone all related to various positive 
extracurricular writing activities, suggesting the factar may give some 
Indlcdtlon of the extent and variety of writing the student does outside of 
school. One negative item which loaded negatively on tfiis factor also loaded 

-4- 6 



negatively on Factor 3 which wcs In contrast to all the negative loadings of 
positive Items on that factor. Factor 2 single loadings all suggested pride 
In writing, as did an Item which loaded on both Factor 2 and Factor 3. The 
Stronger loading of this Item on Factor 3 suggested Its Inclusion there. The 
^Ird factor Item loadings were all «ltems which Indicate displeasure ^Ith 
writing, Insecu^'^'ty about writing, and generally an Indication of writing 
self-concept. The double loading Item which loaded more strongly here than on 
F'actor 2 was a positive Item which loaded negatively as would be expected. 
The content of Items loading on Factor 4 did not suggest a tentative label* 
Furthermore, among the four Items loading on this factor, all of which loaded 
positively, one Item had previously been Judged to be a negative Item* 
Suggested factors and Items as well as their loadings are shown In Table 2« 



^ Table 2 

Factor Analysis ^uf Florida Writing Project 
Student Survey Items 



Variable 



Loadings 





I 




III 


IV 


Factor I Extent & Variety of Writing 










Item 1 (+) 


.56 


.18 


-.31 


.10 


Item 2 (-) 


-.41 


.03 


.60 


-.11 


Item 12 (+) 


.73 


.07 


.05 


-.05 


Item 13 (+) 


.74 


.05 


.04 


.04 


Item 15 (+) 


.54 


.19 , 


-.02. 


.11 


Item 16 (+) 


.59 


.10 


-.02 


-.05 


Item 25 (+) 


.62 


.25 


-.15 


.17 

• 


Factor 2 Pride In Writing 




• 






Item 7 (+) 


.19 


.73 


-.08 


-.09 


Item 11 (+) 


.37 


.46 


-.13 


.13 


Item 14 (+) 


.19 


.51 


-.17 




Item 20 (+) 


.08 


.45 


-.•7 


.13 


Item 21 (+) 


.11 


.77 


.01 


-.03 
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Tabl« 2 Continued 



Loadings 



Variable 



< 


I 


II 


T TT 

III 


IV 


Factor 3 Writing Self-Concept 










Item 3 (-) 


• 36 


-.09 


• 53 


-•18 


Item «♦ (-) 


— 11 
li 






• uu 


Item 5 (+) 


.07 


.46 


-.54 


.12 


Item 18 (-^ 


-.01 


-.11 


.53 


.18 


Item 19 (-) 


.06 


-.07 


.62 


.04 


Item 22 (-) 


.10 


-.29 


.^5 


.06 


Factor 4 










Item 6 (+) 


-.06 


.00 


.11 


.44 


Item 10 (+) 


.19 


.18 


-.06 


.54 


Item 17 (+) 


.26 


.20 


-.13 


.41 


Items not loading 


ft 








Item & (+) 


.23 


' .33 


-.10 


.38 


Item 9 (-)* 


-.08 


-.13 


.09 


.66 


Item 23 (+) 


.33 


.38 


-.10 


.22 


• Item 24 (+) 


.28 


.21 

O 


-.20 


.25 



*Slgn of loading was not consistent with others on factor • 



Item analysis results showing corrected Item- total correlations are 
provided in Table 3 for pretest (N - 2154), posttest (N - 1896), and combined 
(N « 4057) analyses • Results revealed that in the case of two items (6 and 
9), itemr total correlations were below .10 in all three analyses. For another 
three items (18, 19, and 22), correlations fell between .10 and .29 
consistently. The remaining 20 items had corrected item-total correlations 
ranging from .31 to .59 • While several items have low item-total 
correlations^ the deletion of any one item would not increase coefficient 
alpha appreciably, presumably due to the low number of items Included on the 
scale to begin with. 



Table 3 



Corrected Item-Total Correlations of FU? Student Survey Items 
for Pretest, Posttest, and Both Tests Combined 



Corrected Item-Total Correlations 

Item 



number • 





Pretest (N - 2154) 


Posttest 


(N - 1896) Total (N - 4057) 

t 


1 


.55 


, .55 


•55 




AO 




• 


3 


.^•8 


.52 


.50 


4 


.31 


.33 


.32 


5 


.49 


.53 


.51 


6 


•01 


-.00 


• 

.03 


/ 




AO 


" .46 


8 


.38 


.41 


.39 


9 ' 


-.02 


.00 


-.00 > 


10 


.32 


.29 


.31 


11 


.48 


.52 


.50 


12 


.40 


.40 


..40 


13 


.46 


.46 


.46 


14 


.44 


.47 


.46 


15 


.41 


.42 


.41 


16 


.36 


.35 


.35 


17 


.37 


.36 


.36 


18 


.24 


.23 


.24 


19 


.23 


.27 


.25 


20 


.42 


.44 


.43 


21 


.39 


.40 


.40 


22 


.24 


.26 


..25 


23 


.40 


.46 


.43 


24 


.37 


.37 


.37 


25 


.54 


.59 


i56 


An abbreviated summary of analysis of variance results across all grades 
is provided in Table 4. In only one instance (at grade 0) wias there a 
significant twc^way interaction between measure (pre/post) and group 



( experimental/ inservife / cont rol ) . This result is not surprising since it is 
generally consistent with the absence of positive treatment effect indicated 



In the analysis 'of holistic writing scores • The primary result of Interest 

i 

was the main effect for measure (pretest/posttest) • Here there was a 
significant main effect in six of s&ven grades. The 'one Instance In .which a 
significant main effect for measure was not found was at grade 6. In all 
cases the change was 1^ a positive direction from pretest to posttest* This 
Included the result at grade 8 In which there was an Interaction effect » and 
the simple effects revealed the posttest result for Inservlce students was 
greater than the pretest result • The change from pretest to posttest In the 
absence of a positive effect ,of treatment leads to the speculation that 
changes may have been due to maturity. Nonetheless » the evidence supports the 
notion that the Instrument Is sensitive to change. 

0 

.Table 4 

Summary by Grade of Analysis of Variance Results for 
Florida Writing Project Student Survey 





F 


F 




Grade 


Two-Way 


Main Effects 


Simple Effects/ 




Interaction 


-Measure 


Follow-ups 


6 


.15 


.23 




7 


1.23 


6.92** 


post > pre 


8 


4.89* 


5.89** 


test (3 Inservlce 








post > pre 


9 


.37 


7 .02** 


post > pre 


10 


2.31 


7.23** 


post > pre . 


11 


.39 


20.59** 


post > pre 


12 


.74 


8.00** 


post > pre ^ 



* p < .05 
.** p < .01. 



Results of pretest, posttest, and gain score correlations of the attitude 
scores with holistic scores are shown In Table 5. Significant correlations 
were found on pretest scores at 6 of 7 grades with significant correlations 
ranging from .23 to^.50. Posttest correlations were also significant for 6 of 



ERIC 



8-10 



the 7 grades and ranged from^ «20 to •SS. There was only one significant 
correlation for gain scores. Despite the number of statls':lcally significant 
correlations, the magnitude of the relationships in even the strongest 
relationships Is moderate at best. Most relationships are weak. 

0 

Table 5 

Correlations of Holistic Scores with Attitude Scores at Each Grade 



-c 






V 


1 


Grade 


N ' 


Pretest 


Posttest 


Gain 


6 


50 


.24* 


.2A* 


. 14 


7 


62 


.50** 


.36** 


.27* 


8 


72. 


.12 


.28** 


.02 


9 


75 


.31** 


.08 


-.04 


10 


108 


.23* 


.38** 


.11 


11 


116 


.23** 


.20* 


.01 


12 


107 


.34** 

a- 


.31** 


.02 



*p < .05 • 
**p < .01 



Summary and Reconmendatlons 
A total of 25 Llkert-'type Items rated on a S«»polnt scale comprise a 
measure of student attitudes toward vnrltlng. While the Internal consistency 
of the scale was judged to be adequate for research purposes , test-tetest 
reliabilities were somewhat low. Length of time between pretest and posttest 
may have contr4.buted to the low obtained values. Factor analysis revealed 
three fairly well defined factors^ with a fourth factor which was less well 
defined and three Items which did not load on any factor* Item analysis 



9- 11 



revealed five Items which require further scrutiny and possible revision or 

elimination* Analysis of variance results Indicate the Instrument is 

sensitive to change « Correlations of the attitude scale with holistic scores, 

while 8lgi:lf leant, were generally weak. 

Further analyses are recommended as a means of refining the survey and 

providing additional validation Information. Item discrimination *as well as 

t 

examination of the ^l*:ems with low Item^total correlations would aid In 
decisions to revise or remove Items* This should be done In conjunction with 
an examination of the effect of Item removal on coefficient alpha since the 
number of item^ Is already small* Test-retest reliabilities .should also be 
reexamined using shorter time Intervals between testings* If the use of 
factors Is desired, analyses already^ completed should be performed on factor 
scores* Finally, , correlations of the scale with criterion measures would aid 
In providing evidence of content validity* 



12 
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FLORIDA WRITING PROJECT STUDENT v.^VEY 



Key: 


y agree 


0) 

z 

C7) C/l 
rd 

•o 

^ oLi >> 




I strongl 


I agree j 
I sometiiil 
i Qisagre 
I strongl 


Sample ^ 

Ex. 1. My favorite subject In school Is biology. 


1 1 • I i 
A B C D E 

Ex.l.O#®©® 



Directions: Indicate your agreement or disagreement with the following comments 
by marking your answers In pencil on the computer sheet. 
DO NOT WRITE ON THIS PAGE. 



\.r I write for relaxation or as a hobby. 

2. I have to force myself to write. 

3. Writing is one of the activities I like least in school. 

4. I have difficulty beginning a writing assignment. 

5. I am a good writer. 

6. Good writers spend more time than poor writers in revising their work. 

7. I share my writing with others. 

8. I revise my writing to make it better. 

9. The teacher is the most Important audience for what I write in school. 
10. In general, I like school, 
n. I save my writing. 

12. I write notes to my family and friends. 

13. I write letters • 

14. I am proud of at least one piece of writing I have written during the last year 

15. I am sometimes able to write about things that are hard for me to say. 

16. I keep a journal or a diary. e 

17. I enjoy reading. . ''^ 

18. I have good ideas, but I can't put them down on paper. 

19. I make too many mechanical errors when I write. 

20. At least one teacher I have had during my years in school has told me that 

I am a good writer. 

21. In class. I share what I write with other students. 

22. I am embarrassed by my writing. 

23. r have many stories I would like to tell in writing. 

24. Writing will probably be a part of the job I plan to hold in the future. 

25. Writing is an important way for me to express my feelings. 

Florida Writing Project - H.L Quttinger, CM. Morris, 1983 



